{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 深度残差网络（ResNet）\n",
    "\n",
    "ResNet是微软亚洲研究院在ImageNet大规模视觉识别挑战赛2015（ILSVRC14）上使用的一种网络架构，最终在比赛中Top-5 error为3.57%,从而获得当年比赛第一名。该网络架构主要设动机的是为了解决在网络深度增加的情况下精度退化（精度下降）的问题，该问题的原因之一是梯度消失/爆炸。\n",
    "\n",
    "## Deeper Bottleneck Architectures\n",
    "\n",
    "ResNet设计者通过实验发现通常需要直接求解的映射H(x)难以优化，而H(x)的残差形式 F(x) = H(x) - x 易于优化，所以得到如下图所示的残差网络的形式。\n",
    "![](img/res_block.PNG)\n",
    "> 图片来源：[Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385v1.pdf)\n",
    "\n",
    "基于以上形式，在实现ResNet时采用如下图的Deeper Bottleneck Architectures不断进行堆叠。\n",
    "![](img/DBA.PNG)\n",
    "> 图片来源：[Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385v1.pdf)\n",
    "\n",
    "## 恒等映射\n",
    "\n",
    "在DBA的架构中，选择不同的残差结构会对最终结果产生不同的影响，最终的实验证明，恒等映射是最佳实现。首先将残差形式变换为 y = F(x) + h(x)，其中的 h(x) 为shortcut连接。\n",
    "- 1、关于shortcut的实验，采用不同的shortcut对最终的结果影响如下图所示。\n",
    "![](img/shortcut.PNG)\n",
    "> 图片来源：[Identity Mappings in Deep Residual Networks](https://arxiv.org/pdf/1603.05027v2.pdf)\n",
    "\n",
    "![](img/shortcut_res.PNG)\n",
    "> 图片来源：[Identity Mappings in Deep Residual Networks](https://arxiv.org/pdf/1603.05027v2.pdf)\n",
    "\n",
    "结果表明 h(x) = x 时结果最优。\n",
    "- 2、关于activation的实验，将不同的activation放在DBA不同的位置对最终结果影响如下图所示。\n",
    "![](img/activation.PNG)\n",
    "> 图片来源：[Identity Mappings in Deep Residual Networks](https://arxiv.org/pdf/1603.05027v2.pdf)\n",
    "\n",
    "结果表明在卷积操作前加入BN和ReLU激活操作时结果最优。\n",
    "\n",
    "## 网络架构\n",
    "\n",
    "ResNet的完整网络架构如下图所示：\n",
    "![](img/resnet.PNG)\n",
    "> 图片来源：[Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385v1.pdf)\n",
    "\n",
    "以50-layer为例，进入DBA层之前的网络为：①CONV层\"7×7, 64, stride 2\"、②BN层、③ReLU层、④池化层\"3×3 max pool, stride 2\"，最终的输出结果是一个大小为 [batch_size, height, width, kernels] 矩阵。\n",
    "\n",
    "后续是16个相似或者重复的DBA网络，具体实现类似下图所示：\n",
    "![](img/DBA_implemention.PNG)\n",
    "> 图片来源：[Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385v1.pdf)\n",
    "\n",
    "堆叠的DBA网络之后直接连接①average pool、②1000-d fc、③softmax，直接输出结果。\n",
    "\n",
    "## 代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from collections import namedtuple\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import six\n",
    "\n",
    "from tensorflow.python.training import moving_averages\n",
    "\n",
    "\n",
    "HParams = namedtuple('HParams',\n",
    "                     'batch_size, num_classes, min_lrn_rate, lrn_rate, '\n",
    "                     'num_residual_units, use_bottleneck, weight_decay_rate, '\n",
    "                     'relu_leakiness, optimizer')\n",
    "\n",
    "\n",
    "class ResNet(object):\n",
    "  \"\"\"ResNet model.\"\"\"\n",
    "\n",
    "  def __init__(self, hps, images, labels, mode):\n",
    "    \"\"\"ResNet constructor.\n",
    "\n",
    "    Args:\n",
    "      hps: Hyperparameters.\n",
    "      images: Batches of images. [batch_size, image_size, image_size, 3]\n",
    "      labels: Batches of labels. [batch_size, num_classes]\n",
    "      mode: One of 'train' and 'eval'.\n",
    "    \"\"\"\n",
    "    self.hps = hps\n",
    "    self._images = images\n",
    "    self.labels = labels\n",
    "    self.mode = mode\n",
    "\n",
    "    self._extra_train_ops = []\n",
    "\n",
    "  def build_graph(self):\n",
    "    \"\"\"Build a whole graph for the model.\"\"\"\n",
    "    self.global_step = tf.train.get_or_create_global_step()\n",
    "    self._build_model()\n",
    "    if self.mode == 'train':\n",
    "      self._build_train_op()\n",
    "    self.summaries = tf.summary.merge_all()\n",
    "\n",
    "  def _stride_arr(self, stride):\n",
    "    \"\"\"Map a stride scalar to the stride array for tf.nn.conv2d.\"\"\"\n",
    "    return [1, stride, stride, 1]\n",
    "\n",
    "  def _build_model(self):\n",
    "    \"\"\"Build the core model within the graph.\"\"\"\n",
    "    with tf.variable_scope('init'):\n",
    "      x = self._images\n",
    "      x = self._conv('init_conv', x, 3, 3, 16, self._stride_arr(1))\n",
    "\n",
    "    strides = [1, 2, 2]\n",
    "    activate_before_residual = [True, False, False]\n",
    "    if self.hps.use_bottleneck:\n",
    "      res_func = self._bottleneck_residual\n",
    "      filters = [16, 64, 128, 256]\n",
    "    else:\n",
    "      res_func = self._residual\n",
    "      filters = [16, 16, 32, 64]\n",
    "      # Uncomment the following codes to use w28-10 wide residual network.\n",
    "      # It is more memory efficient than very deep residual network and has\n",
    "      # comparably good performance.\n",
    "      # https://arxiv.org/pdf/1605.07146v1.pdf\n",
    "      # filters = [16, 160, 320, 640]\n",
    "      # Update hps.num_residual_units to 4\n",
    "\n",
    "    with tf.variable_scope('unit_1_0'):\n",
    "      x = res_func(x, filters[0], filters[1], self._stride_arr(strides[0]),\n",
    "                   activate_before_residual[0])\n",
    "    for i in six.moves.range(1, self.hps.num_residual_units):\n",
    "      with tf.variable_scope('unit_1_%d' % i):\n",
    "        x = res_func(x, filters[1], filters[1], self._stride_arr(1), False)\n",
    "\n",
    "    with tf.variable_scope('unit_2_0'):\n",
    "      x = res_func(x, filters[1], filters[2], self._stride_arr(strides[1]),\n",
    "                   activate_before_residual[1])\n",
    "    for i in six.moves.range(1, self.hps.num_residual_units):\n",
    "      with tf.variable_scope('unit_2_%d' % i):\n",
    "        x = res_func(x, filters[2], filters[2], self._stride_arr(1), False)\n",
    "\n",
    "    with tf.variable_scope('unit_3_0'):\n",
    "      x = res_func(x, filters[2], filters[3], self._stride_arr(strides[2]),\n",
    "                   activate_before_residual[2])\n",
    "    for i in six.moves.range(1, self.hps.num_residual_units):\n",
    "      with tf.variable_scope('unit_3_%d' % i):\n",
    "        x = res_func(x, filters[3], filters[3], self._stride_arr(1), False)\n",
    "\n",
    "    with tf.variable_scope('unit_last'):\n",
    "      x = self._batch_norm('final_bn', x)\n",
    "      x = self._relu(x, self.hps.relu_leakiness)\n",
    "      x = self._global_avg_pool(x)\n",
    "\n",
    "    with tf.variable_scope('logit'):\n",
    "      logits = self._fully_connected(x, self.hps.num_classes)\n",
    "      self.predictions = tf.nn.softmax(logits)\n",
    "\n",
    "    with tf.variable_scope('costs'):\n",
    "      xent = tf.nn.softmax_cross_entropy_with_logits(\n",
    "          logits=logits, labels=self.labels)\n",
    "      self.cost = tf.reduce_mean(xent, name='xent')\n",
    "      self.cost += self._decay()\n",
    "\n",
    "      tf.summary.scalar('cost', self.cost)\n",
    "\n",
    "  def _build_train_op(self):\n",
    "    \"\"\"Build training specific ops for the graph.\"\"\"\n",
    "    self.lrn_rate = tf.constant(self.hps.lrn_rate, tf.float32)\n",
    "    tf.summary.scalar('learning_rate', self.lrn_rate)\n",
    "\n",
    "    trainable_variables = tf.trainable_variables()\n",
    "    grads = tf.gradients(self.cost, trainable_variables)\n",
    "\n",
    "    if self.hps.optimizer == 'sgd':\n",
    "      optimizer = tf.train.GradientDescentOptimizer(self.lrn_rate)\n",
    "    elif self.hps.optimizer == 'mom':\n",
    "      optimizer = tf.train.MomentumOptimizer(self.lrn_rate, 0.9)\n",
    "\n",
    "    apply_op = optimizer.apply_gradients(\n",
    "        zip(grads, trainable_variables),\n",
    "        global_step=self.global_step, name='train_step')\n",
    "\n",
    "    train_ops = [apply_op] + self._extra_train_ops\n",
    "    self.train_op = tf.group(*train_ops)\n",
    "\n",
    "  # TODO(xpan): Consider batch_norm in contrib/layers/python/layers/layers.py\n",
    "  def _batch_norm(self, name, x):\n",
    "    \"\"\"Batch normalization.\"\"\"\n",
    "    with tf.variable_scope(name):\n",
    "      params_shape = [x.get_shape()[-1]]\n",
    "\n",
    "      beta = tf.get_variable(\n",
    "          'beta', params_shape, tf.float32,\n",
    "          initializer=tf.constant_initializer(0.0, tf.float32))\n",
    "      gamma = tf.get_variable(\n",
    "          'gamma', params_shape, tf.float32,\n",
    "          initializer=tf.constant_initializer(1.0, tf.float32))\n",
    "\n",
    "      if self.mode == 'train':\n",
    "        mean, variance = tf.nn.moments(x, [0, 1, 2], name='moments')\n",
    "\n",
    "        moving_mean = tf.get_variable(\n",
    "            'moving_mean', params_shape, tf.float32,\n",
    "            initializer=tf.constant_initializer(0.0, tf.float32),\n",
    "            trainable=False)\n",
    "        moving_variance = tf.get_variable(\n",
    "            'moving_variance', params_shape, tf.float32,\n",
    "            initializer=tf.constant_initializer(1.0, tf.float32),\n",
    "            trainable=False)\n",
    "\n",
    "        self._extra_train_ops.append(moving_averages.assign_moving_average(\n",
    "            moving_mean, mean, 0.9))\n",
    "        self._extra_train_ops.append(moving_averages.assign_moving_average(\n",
    "            moving_variance, variance, 0.9))\n",
    "      else:\n",
    "        mean = tf.get_variable(\n",
    "            'moving_mean', params_shape, tf.float32,\n",
    "            initializer=tf.constant_initializer(0.0, tf.float32),\n",
    "            trainable=False)\n",
    "        variance = tf.get_variable(\n",
    "            'moving_variance', params_shape, tf.float32,\n",
    "            initializer=tf.constant_initializer(1.0, tf.float32),\n",
    "            trainable=False)\n",
    "        tf.summary.histogram(mean.op.name, mean)\n",
    "        tf.summary.histogram(variance.op.name, variance)\n",
    "      # epsilon used to be 1e-5. Maybe 0.001 solves NaN problem in deeper net.\n",
    "      y = tf.nn.batch_normalization(\n",
    "          x, mean, variance, beta, gamma, 0.001)\n",
    "      y.set_shape(x.get_shape())\n",
    "      return y\n",
    "\n",
    "  def _residual(self, x, in_filter, out_filter, stride,\n",
    "                activate_before_residual=False):\n",
    "    \"\"\"Residual unit with 2 sub layers.\"\"\"\n",
    "    if activate_before_residual:\n",
    "      with tf.variable_scope('shared_activation'):\n",
    "        x = self._batch_norm('init_bn', x)\n",
    "        x = self._relu(x, self.hps.relu_leakiness)\n",
    "        orig_x = x\n",
    "    else:\n",
    "      with tf.variable_scope('residual_only_activation'):\n",
    "        orig_x = x\n",
    "        x = self._batch_norm('init_bn', x)\n",
    "        x = self._relu(x, self.hps.relu_leakiness)\n",
    "\n",
    "    with tf.variable_scope('sub1'):\n",
    "      x = self._conv('conv1', x, 3, in_filter, out_filter, stride)\n",
    "\n",
    "    with tf.variable_scope('sub2'):\n",
    "      x = self._batch_norm('bn2', x)\n",
    "      x = self._relu(x, self.hps.relu_leakiness)\n",
    "      x = self._conv('conv2', x, 3, out_filter, out_filter, [1, 1, 1, 1])\n",
    "\n",
    "    with tf.variable_scope('sub_add'):\n",
    "      if in_filter != out_filter:\n",
    "        orig_x = tf.nn.avg_pool(orig_x, stride, stride, 'VALID')\n",
    "        orig_x = tf.pad(\n",
    "            orig_x, [[0, 0], [0, 0], [0, 0],\n",
    "                     [(out_filter-in_filter)//2, (out_filter-in_filter)//2]])\n",
    "      x += orig_x\n",
    "\n",
    "    tf.logging.debug('image after unit %s', x.get_shape())\n",
    "    return x\n",
    "\n",
    "  def _bottleneck_residual(self, x, in_filter, out_filter, stride,\n",
    "                           activate_before_residual=False):\n",
    "    \"\"\"Bottleneck residual unit with 3 sub layers.\"\"\"\n",
    "    if activate_before_residual:\n",
    "      with tf.variable_scope('common_bn_relu'):\n",
    "        x = self._batch_norm('init_bn', x)\n",
    "        x = self._relu(x, self.hps.relu_leakiness)\n",
    "        orig_x = x\n",
    "    else:\n",
    "      with tf.variable_scope('residual_bn_relu'):\n",
    "        orig_x = x\n",
    "        x = self._batch_norm('init_bn', x)\n",
    "        x = self._relu(x, self.hps.relu_leakiness)\n",
    "\n",
    "    with tf.variable_scope('sub1'):\n",
    "      x = self._conv('conv1', x, 1, in_filter, out_filter/4, stride)\n",
    "\n",
    "    with tf.variable_scope('sub2'):\n",
    "      x = self._batch_norm('bn2', x)\n",
    "      x = self._relu(x, self.hps.relu_leakiness)\n",
    "      x = self._conv('conv2', x, 3, out_filter/4, out_filter/4, [1, 1, 1, 1])\n",
    "\n",
    "    with tf.variable_scope('sub3'):\n",
    "      x = self._batch_norm('bn3', x)\n",
    "      x = self._relu(x, self.hps.relu_leakiness)\n",
    "      x = self._conv('conv3', x, 1, out_filter/4, out_filter, [1, 1, 1, 1])\n",
    "\n",
    "    with tf.variable_scope('sub_add'):\n",
    "      if in_filter != out_filter:\n",
    "        orig_x = self._conv('project', orig_x, 1, in_filter, out_filter, stride)\n",
    "      x += orig_x\n",
    "\n",
    "    tf.logging.info('image after unit %s', x.get_shape())\n",
    "    return x\n",
    "\n",
    "  def _decay(self):\n",
    "    \"\"\"L2 weight decay loss.\"\"\"\n",
    "    costs = []\n",
    "    for var in tf.trainable_variables():\n",
    "      if var.op.name.find(r'DW') > 0:\n",
    "        costs.append(tf.nn.l2_loss(var))\n",
    "        # tf.summary.histogram(var.op.name, var)\n",
    "\n",
    "    return tf.multiply(self.hps.weight_decay_rate, tf.add_n(costs))\n",
    "\n",
    "  def _conv(self, name, x, filter_size, in_filters, out_filters, strides):\n",
    "    \"\"\"Convolution.\"\"\"\n",
    "    with tf.variable_scope(name):\n",
    "      n = filter_size * filter_size * out_filters\n",
    "      kernel = tf.get_variable(\n",
    "          'DW', [filter_size, filter_size, in_filters, out_filters],\n",
    "          tf.float32, initializer=tf.random_normal_initializer(\n",
    "              stddev=np.sqrt(2.0/n)))\n",
    "      return tf.nn.conv2d(x, kernel, strides, padding='SAME')\n",
    "\n",
    "  def _relu(self, x, leakiness=0.0):\n",
    "    \"\"\"Relu, with optional leaky support.\"\"\"\n",
    "    return tf.where(tf.less(x, 0.0), leakiness * x, x, name='leaky_relu')\n",
    "\n",
    "  def _fully_connected(self, x, out_dim):\n",
    "    \"\"\"FullyConnected layer for final output.\"\"\"\n",
    "    x = tf.reshape(x, [self.hps.batch_size, -1])\n",
    "    w = tf.get_variable(\n",
    "        'DW', [x.get_shape()[1], out_dim],\n",
    "        initializer=tf.uniform_unit_scaling_initializer(factor=1.0))\n",
    "    b = tf.get_variable('biases', [out_dim],\n",
    "                        initializer=tf.constant_initializer())\n",
    "    return tf.nn.xw_plus_b(x, w, b)\n",
    "\n",
    "  def _global_avg_pool(self, x):\n",
    "    assert x.get_shape().ndims == 4\n",
    "    return tf.reduce_mean(x, [1, 2])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在cifar-10上训练模型\n",
    "\n",
    "数据准备：在运行服务器上下载cifar-10数据，将解压后的数据拷贝到当前目录下的cifar10文件夹下，下载可以使用如下命令：\n",
    "```shell\n",
    "curl -o cifar-10-binary.tar.gz https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import six\n",
    "import sys\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "FLAGS = tf.app.flags.FLAGS\n",
    "tf.app.flags.DEFINE_string('dataset', 'cifar10', 'cifar10 or cifar100.')\n",
    "tf.app.flags.DEFINE_string('mode', 'train', 'train or eval.')\n",
    "tf.app.flags.DEFINE_string('train_data_path', 'images/cifar10/data_batch*',\n",
    "                           'Filepattern for training data.')\n",
    "tf.app.flags.DEFINE_string('eval_data_path', '',\n",
    "                           'Filepattern for eval data')\n",
    "tf.app.flags.DEFINE_integer('image_size', 32, 'Image side length.')\n",
    "tf.app.flags.DEFINE_string('train_dir', '/tmp/resnet_model/train',\n",
    "                           'Directory to keep training outputs.')\n",
    "tf.app.flags.DEFINE_string('eval_dir', '',\n",
    "                           'Directory to keep eval outputs.')\n",
    "tf.app.flags.DEFINE_integer('eval_batch_count', 50,\n",
    "                            'Number of batches to eval.')\n",
    "tf.app.flags.DEFINE_bool('eval_once', False,\n",
    "                         'Whether evaluate the model only once.')\n",
    "tf.app.flags.DEFINE_string('log_root', '/tmp/resnet_model',\n",
    "                           'Directory to keep the checkpoints. Should be a '\n",
    "                           'parent directory of FLAGS.train_dir/eval_dir.')\n",
    "tf.app.flags.DEFINE_integer('num_gpus', 1,\n",
    "                            'Number of gpus used for training. (0 or 1)')\n",
    "\n",
    "\n",
    "def train(hps):\n",
    "  \"\"\"Training loop.\"\"\"\n",
    "  images, labels = build_input(\n",
    "      FLAGS.dataset, FLAGS.train_data_path, hps.batch_size, FLAGS.mode)\n",
    "  model = ResNet(hps, images, labels, FLAGS.mode)\n",
    "  model.build_graph()\n",
    "\n",
    "  param_stats = tf.contrib.tfprof.model_analyzer.print_model_analysis(\n",
    "      tf.get_default_graph(),\n",
    "      tfprof_options=tf.contrib.tfprof.model_analyzer.\n",
    "          TRAINABLE_VARS_PARAMS_STAT_OPTIONS)\n",
    "  sys.stdout.write('total_params: %d\\n' % param_stats.total_parameters)\n",
    "\n",
    "  tf.contrib.tfprof.model_analyzer.print_model_analysis(\n",
    "      tf.get_default_graph(),\n",
    "      tfprof_options=tf.contrib.tfprof.model_analyzer.FLOAT_OPS_OPTIONS)\n",
    "\n",
    "  truth = tf.argmax(model.labels, axis=1)\n",
    "  predictions = tf.argmax(model.predictions, axis=1)\n",
    "  precision = tf.reduce_mean(tf.to_float(tf.equal(predictions, truth)))\n",
    "\n",
    "  summary_hook = tf.train.SummarySaverHook(\n",
    "      save_steps=100,\n",
    "      output_dir=FLAGS.train_dir,\n",
    "      summary_op=tf.summary.merge([model.summaries,\n",
    "                                   tf.summary.scalar('Precision', precision)]))\n",
    "\n",
    "  logging_hook = tf.train.LoggingTensorHook(\n",
    "      tensors={'step': model.global_step,\n",
    "               'loss': model.cost,\n",
    "               'precision': precision},\n",
    "      every_n_iter=100)\n",
    "\n",
    "  class _LearningRateSetterHook(tf.train.SessionRunHook):\n",
    "    \"\"\"Sets learning_rate based on global step.\"\"\"\n",
    "\n",
    "    def begin(self):\n",
    "      self._lrn_rate = 0.1\n",
    "\n",
    "    def before_run(self, run_context):\n",
    "      return tf.train.SessionRunArgs(\n",
    "          model.global_step,  # Asks for global step value.\n",
    "          feed_dict={model.lrn_rate: self._lrn_rate})  # Sets learning rate\n",
    "\n",
    "    def after_run(self, run_context, run_values):\n",
    "      train_step = run_values.results\n",
    "      if train_step < 40000:\n",
    "        self._lrn_rate = 0.1\n",
    "      elif train_step < 60000:\n",
    "        self._lrn_rate = 0.01\n",
    "      elif train_step < 80000:\n",
    "        self._lrn_rate = 0.001\n",
    "      else:\n",
    "        self._lrn_rate = 0.0001\n",
    "\n",
    "  with tf.train.MonitoredTrainingSession(\n",
    "      checkpoint_dir=FLAGS.log_root,\n",
    "      hooks=[logging_hook, _LearningRateSetterHook()],\n",
    "      chief_only_hooks=[summary_hook],\n",
    "      # Since we provide a SummarySaverHook, we need to disable default\n",
    "      # SummarySaverHook. To do that we set save_summaries_steps to 0.\n",
    "      save_summaries_steps=0,\n",
    "      config=tf.ConfigProto(allow_soft_placement=True)) as mon_sess:\n",
    "    while not mon_sess.should_stop():\n",
    "      mon_sess.run(model.train_op)\n",
    "\n",
    "\n",
    "def evaluate(hps):\n",
    "  \"\"\"Eval loop.\"\"\"\n",
    "  images, labels = build_input(\n",
    "      FLAGS.dataset, FLAGS.eval_data_path, hps.batch_size, FLAGS.mode)\n",
    "  model = ResNet(hps, images, labels, FLAGS.mode)\n",
    "  model.build_graph()\n",
    "  saver = tf.train.Saver()\n",
    "  summary_writer = tf.summary.FileWriter(FLAGS.eval_dir)\n",
    "\n",
    "  sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True))\n",
    "  tf.train.start_queue_runners(sess)\n",
    "\n",
    "  best_precision = 0.0\n",
    "  while True:\n",
    "    try:\n",
    "      ckpt_state = tf.train.get_checkpoint_state(FLAGS.log_root)\n",
    "    except tf.errors.OutOfRangeError as e:\n",
    "      tf.logging.error('Cannot restore checkpoint: %s', e)\n",
    "      continue\n",
    "    if not (ckpt_state and ckpt_state.model_checkpoint_path):\n",
    "      tf.logging.info('No model to eval yet at %s', FLAGS.log_root)\n",
    "      continue\n",
    "    tf.logging.info('Loading checkpoint %s', ckpt_state.model_checkpoint_path)\n",
    "    saver.restore(sess, ckpt_state.model_checkpoint_path)\n",
    "\n",
    "    total_prediction, correct_prediction = 0, 0\n",
    "    for _ in six.moves.range(FLAGS.eval_batch_count):\n",
    "      (summaries, loss, predictions, truth, train_step) = sess.run(\n",
    "          [model.summaries, model.cost, model.predictions,\n",
    "           model.labels, model.global_step])\n",
    "\n",
    "      truth = np.argmax(truth, axis=1)\n",
    "      predictions = np.argmax(predictions, axis=1)\n",
    "      correct_prediction += np.sum(truth == predictions)\n",
    "      total_prediction += predictions.shape[0]\n",
    "\n",
    "    precision = 1.0 * correct_prediction / total_prediction\n",
    "    best_precision = max(precision, best_precision)\n",
    "\n",
    "    precision_summ = tf.Summary()\n",
    "    precision_summ.value.add(\n",
    "        tag='Precision', simple_value=precision)\n",
    "    summary_writer.add_summary(precision_summ, train_step)\n",
    "    best_precision_summ = tf.Summary()\n",
    "    best_precision_summ.value.add(\n",
    "        tag='Best Precision', simple_value=best_precision)\n",
    "    summary_writer.add_summary(best_precision_summ, train_step)\n",
    "    summary_writer.add_summary(summaries, train_step)\n",
    "    tf.logging.info('loss: %.3f, precision: %.3f, best precision: %.3f' %\n",
    "                    (loss, precision, best_precision))\n",
    "    summary_writer.flush()\n",
    "\n",
    "    if FLAGS.eval_once:\n",
    "      break\n",
    "\n",
    "    time.sleep(60)\n",
    "\n",
    "#cifar-10 data preprocessing\n",
    "def build_input(dataset, data_path, batch_size, mode):\n",
    "  \"\"\"Build CIFAR image and labels.\n",
    "\n",
    "  Args:\n",
    "    dataset: Either 'cifar10' or 'cifar100'.\n",
    "    data_path: Filename for data.\n",
    "    batch_size: Input batch size.\n",
    "    mode: Either 'train' or 'eval'.\n",
    "  Returns:\n",
    "    images: Batches of images. [batch_size, image_size, image_size, 3]\n",
    "    labels: Batches of labels. [batch_size, num_classes]\n",
    "  Raises:\n",
    "    ValueError: when the specified dataset is not supported.\n",
    "  \"\"\"\n",
    "  image_size = 32\n",
    "  if dataset == 'cifar10':\n",
    "    label_bytes = 1\n",
    "    label_offset = 0\n",
    "    num_classes = 10\n",
    "  elif dataset == 'cifar100':\n",
    "    label_bytes = 1\n",
    "    label_offset = 1\n",
    "    num_classes = 100\n",
    "  else:\n",
    "    raise ValueError('Not supported dataset %s', dataset)\n",
    "\n",
    "  depth = 3\n",
    "  image_bytes = image_size * image_size * depth\n",
    "  record_bytes = label_bytes + label_offset + image_bytes\n",
    "\n",
    "  data_files = tf.gfile.Glob(data_path)\n",
    "  file_queue = tf.train.string_input_producer(data_files, shuffle=True)\n",
    "  # Read examples from files in the filename queue.\n",
    "  reader = tf.FixedLengthRecordReader(record_bytes=record_bytes)\n",
    "  _, value = reader.read(file_queue)\n",
    "\n",
    "  # Convert these examples to dense labels and processed images.\n",
    "  record = tf.reshape(tf.decode_raw(value, tf.uint8), [record_bytes])\n",
    "  label = tf.cast(tf.slice(record, [label_offset], [label_bytes]), tf.int32)\n",
    "  # Convert from string to [depth * height * width] to [depth, height, width].\n",
    "  depth_major = tf.reshape(tf.slice(record, [label_offset + label_bytes], [image_bytes]),\n",
    "                           [depth, image_size, image_size])\n",
    "  # Convert from [depth, height, width] to [height, width, depth].\n",
    "  image = tf.cast(tf.transpose(depth_major, [1, 2, 0]), tf.float32)\n",
    "\n",
    "  if mode == 'train':\n",
    "    image = tf.image.resize_image_with_crop_or_pad(\n",
    "        image, image_size+4, image_size+4)\n",
    "    image = tf.random_crop(image, [image_size, image_size, 3])\n",
    "    image = tf.image.random_flip_left_right(image)\n",
    "    # Brightness/saturation/constrast provides small gains .2%~.5% on cifar.\n",
    "    # image = tf.image.random_brightness(image, max_delta=63. / 255.)\n",
    "    # image = tf.image.random_saturation(image, lower=0.5, upper=1.5)\n",
    "    # image = tf.image.random_contrast(image, lower=0.2, upper=1.8)\n",
    "    image = tf.image.per_image_standardization(image)\n",
    "\n",
    "    example_queue = tf.RandomShuffleQueue(\n",
    "        capacity=16 * batch_size,\n",
    "        min_after_dequeue=8 * batch_size,\n",
    "        dtypes=[tf.float32, tf.int32],\n",
    "        shapes=[[image_size, image_size, depth], [1]])\n",
    "    num_threads = 16\n",
    "  else:\n",
    "    image = tf.image.resize_image_with_crop_or_pad(\n",
    "        image, image_size, image_size)\n",
    "    image = tf.image.per_image_standardization(image)\n",
    "\n",
    "    example_queue = tf.FIFOQueue(\n",
    "        3 * batch_size,\n",
    "        dtypes=[tf.float32, tf.int32],\n",
    "        shapes=[[image_size, image_size, depth], [1]])\n",
    "    num_threads = 1\n",
    "\n",
    "  example_enqueue_op = example_queue.enqueue([image, label])\n",
    "  tf.train.add_queue_runner(tf.train.queue_runner.QueueRunner(\n",
    "      example_queue, [example_enqueue_op] * num_threads))\n",
    "\n",
    "  # Read 'batch' labels + images from the example queue.\n",
    "  images, labels = example_queue.dequeue_many(batch_size)\n",
    "  labels = tf.reshape(labels, [batch_size, 1])\n",
    "  indices = tf.reshape(tf.range(0, batch_size, 1), [batch_size, 1])\n",
    "  labels = tf.sparse_to_dense(\n",
    "      tf.concat(values=[indices, labels], axis=1),\n",
    "      [batch_size, num_classes], 1.0, 0.0)\n",
    "\n",
    "  assert len(images.get_shape()) == 4\n",
    "  assert images.get_shape()[0] == batch_size\n",
    "  assert images.get_shape()[-1] == 3\n",
    "  assert len(labels.get_shape()) == 2\n",
    "  assert labels.get_shape()[0] == batch_size\n",
    "  assert labels.get_shape()[1] == num_classes\n",
    "\n",
    "  # Display the training images in the visualizer.\n",
    "  tf.summary.image('images', images)\n",
    "  return images, labels\n",
    "\n",
    "def main(_):\n",
    "  if FLAGS.num_gpus == 0:\n",
    "    dev = '/cpu:0'\n",
    "  elif FLAGS.num_gpus == 1:\n",
    "    dev = '/gpu:0'\n",
    "  else:\n",
    "    raise ValueError('Only support 0 or 1 gpu.')\n",
    "\n",
    "  if FLAGS.mode == 'train':\n",
    "    batch_size = 128\n",
    "  elif FLAGS.mode == 'eval':\n",
    "    batch_size = 100\n",
    "\n",
    "  if FLAGS.dataset == 'cifar10':\n",
    "    num_classes = 10\n",
    "  elif FLAGS.dataset == 'cifar100':\n",
    "    num_classes = 100\n",
    "\n",
    "  hps = HParams(batch_size=batch_size,\n",
    "                             num_classes=num_classes,\n",
    "                             min_lrn_rate=0.0001,\n",
    "                             lrn_rate=0.1,\n",
    "                             num_residual_units=5,\n",
    "                             use_bottleneck=False,\n",
    "                             weight_decay_rate=0.0002,\n",
    "                             relu_leakiness=0.1,\n",
    "                             optimizer='mom')\n",
    "\n",
    "  with tf.device(dev):\n",
    "    if FLAGS.mode == 'train':\n",
    "      train(hps)\n",
    "    elif FLAGS.mode == 'eval':\n",
    "      evaluate(hps)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "  tf.logging.set_verbosity(tf.logging.INFO)\n",
    "  tf.app.run()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练结果\n",
    "\n",
    "使用tensorboard观察训练过程，如下图：\n",
    "![](img/cifar10.PNG)\n",
    "\n",
    "在训练step在0-40K时learning_rate设置为0.1，precision在0.9左右震荡，cost在0.4左右震荡。\n",
    "\n",
    "在训练step在40K-60K时learning_rate调整为0.01，precision在0.97左右震荡，cost在0.15左右震荡。\n",
    "\n",
    "在训练step在60K-80K时learning_rate调整为0.001，precision在0.99左右，cost在0.1左右,此时训练已趋于收敛。\n",
    "\n",
    "\n",
    "## 参考文献\n",
    "\n",
    "<b>相关论文</b>\n",
    "\n",
    "Identity Mappings in Deep Residual Networks\n",
    "\n",
    "https://arxiv.org/pdf/1603.05027v2.pdf\n",
    "\n",
    "Deep Residual Learning for Image Recognition\n",
    "\n",
    "https://arxiv.org/pdf/1512.03385v1.pdf\n",
    "\n",
    "Wide Residual Networks\n",
    "\n",
    "https://arxiv.org/pdf/1605.07146v1.pdf\n",
    "\n",
    "<b>GitHub地址</b>\n",
    "https://github.com/KaimingHe/deep-residual-networks\n",
    "\n",
    "<b>tensorflow实现</b>\n",
    "https://github.com/tensorflow/models\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
